Repository navigation
feat(review): close remaining review gaps: action dbt_prepare, data-test adequacy, parse-only column changes - #1726
RaghwendraSingh wants to merge 4 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
full receipts (1 session)
orchestrator ·
|
| subagent | cost |
|---|---|
| Goal: an evidence-backed, cited landscape of (a) dbt PR review tools, (b) data-… | 2,134,743 tokens |
| Goal: determine what gaps altimate-code's dbt PR review has, by comparing what… | 5,303,299 tokens |
| Label dbt PR review comments by reading each one. Instructions: read /private/t… | 2,794,842 tokens |
| Label dbt PR review comments by reading each one. Instructions: read /private/t… | 3,244,430 tokens |
| Label escaped-defect candidates by READING them (no keyword bucketing). Directo… | 2,570,116 tokens |
| Repo: /Users/raghwendrasingh/code/altimateai/altimate-code (current branch feat… | 4,836,719 tokens |
| You are doing an ADVERSARIAL review of a set of Jira tickets that plan improvem… | 763,761 tokens |
| Label dbt PR review comments by reading each one. Instructions: read /private/t… | 3,351,198 tokens |
| Label escaped-defect candidates by READING them (no keyword bucketing). Directo… | 4,214,327 tokens |
| Label dbt PR review comments by reading each one. Instructions: read /private/t… | 2,821,134 tokens |
| Label dbt PR review comments by reading each one. Instructions: read /private/t… | 1,795,004 tokens |
| You are doing an ADVERSARIAL review of a stack of changes to an open-source dbt… | 4,084,798 tokens |
| Repo: /Users/raghwendrasingh/code/altimateai/altimate-code. Module: packages/op… | 8,815,483 tokens |
| Label dbt PR review comments by reading each one. Instructions: read /private/t… | 2,592,769 tokens |
| Research task, read-only (do not push, comment, or open PRs anywhere).Goal: dec… | 1,198,872 tokens |
| Judge each of altimate-code's own inline PR review comments by reading it. Inpu… | 1,673,212 tokens |
| Label escaped-defect candidates by READING them (no keyword bucketing). Directo… | 2,862,492 tokens |
| Goal: read a Notion doc tree about an "agentic data engineering software factor… | 1,335,636 tokens |
| Goal: test whether the bug classes that real reviewers catch on dbt PRs at one… | 6,608,416 tokens |
Generated by aireceipts
There was a problem hiding this comment.
10 issues found across 10 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/test/altimate/review-test-adequacy.test.ts">
<violation number="1" location="packages/opencode/test/altimate/review-test-adequacy.test.ts:56">
P3: The test name claims 'deeper wins', but no case sets the same key at two levels. Add a fixture where a folder overrides the project-level `+severity` (for example, `shop: +severity: error` with `marts: +severity: warn`) and assert the deeper value and its path.</violation>
</file>
<file name="packages/opencode/src/altimate/review/test-adequacy.ts">
<violation number="1" location="packages/opencode/src/altimate/review/test-adequacy.ts:28">
P2: `fewer_rows_than` is in `SELF_COMPARE`, but a self-comparison makes it fail every time, not pass. The message "passes by construction" is wrong for this test. Drop it from the set and report it separately, or change the message to cover both outcomes.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/test-adequacy.ts:69">
P2: `refsEntity` matches `ref(` inside names such as `my_ref(`, creating false self-comparison findings. Require a word boundary before `ref` and allow normal whitespace before `(`.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/test-adequacy.ts:117">
P3: `path` records only the last node containing either setting, so findings misidentify the source when `severity` and `error_if` are inherited from different levels. Track a path per threshold.</violation>
<violation number="4" location="packages/opencode/src/altimate/review/test-adequacy.ts:128">
P2: `segmentsOf` fails to strip nested configured model paths, so inherited test thresholds are missed under paths such as `src/dbt/models/marts`. Strip the full configured model directory, not just its first segment.</violation>
<violation number="5" location="packages/opencode/src/altimate/review/test-adequacy.ts:164">
P2: `own` treats any explicit threshold as overriding every inherited setting, so tests can inherit an ineffective `severity` or `error_if` without being reported. Track severity and error thresholds independently.</violation>
</file>
<file name="docs/docs/usage/dbt-pr-review.md">
<violation number="1" location="docs/docs/usage/dbt-pr-review.md:215">
P3: This snippet pins the action to `@main`, but the rest of the page pins `@v0.8.5`. Pin to the same release tag so users get a reviewed, reproducible action version.</violation>
</file>
<file name="packages/opencode/src/altimate/review/orchestrate.ts">
<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:664">
P2: When only the base compile is missing, `columnDelta` now falls through to the raw-SQL path and can emit removed-column warnings. Before this change that case returned no deltas. Confirm this is intended for models marked unverified, or keep returning no deltas when the head is compiled and the base is not.</violation>
</file>
<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">
<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:235">
P2: The `length >= 3` filter drops valid added SQL lines such as `id` or `x`. A multiline window changed only by adding such a two-character expression is then ignored, so the no-`ORDER BY` detector misses the change; filter only empty normalized lines.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/dbt-patterns.ts:243">
P2: The parenthesis scan counts parentheses inside SQL string literals. A value like `status = ')'` closes the window early, truncating `spec` before a real `ORDER BY`; mask quoted literals or use a quote-aware scanner while finding the matching window close.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| "equality", | ||
| "equal_rowcount", | ||
| "cardinality_equality", | ||
| "fewer_rows_than", |
There was a problem hiding this comment.
P2: fewer_rows_than is in SELF_COMPARE, but a self-comparison makes it fail every time, not pass. The message "passes by construction" is wrong for this test. Drop it from the set and report it separately, or change the message to cover both outcomes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/test-adequacy.ts, line 28:
<comment>`fewer_rows_than` is in `SELF_COMPARE`, but a self-comparison makes it fail every time, not pass. The message "passes by construction" is wrong for this test. Drop it from the set and report it separately, or change the message to cover both outcomes.</comment>
<file context>
@@ -0,0 +1,214 @@
+ "equality",
+ "equal_rowcount",
+ "cardinality_equality",
+ "fewer_rows_than",
+ "expect_table_row_count_to_equal_other_table",
+ "expect_table_aggregation_to_equal_other_table",
</file context>
There was a problem hiding this comment.
Fixed in e77d121. fewer_rows_than is dropped (against itself it always fails), and an aggregation comparison on the same model is flagged only when both expressions match.
| current, | ||
| } | ||
| // Compiled SQL on both sides: column lineage plus the final SELECT's names. | ||
| if (engineNewSql && (!withOld || engineOldSql)) { |
There was a problem hiding this comment.
P2: When only the base compile is missing, columnDelta now falls through to the raw-SQL path and can emit removed-column warnings. Before this change that case returned no deltas. Confirm this is intended for models marked unverified, or keep returning no deltas when the head is compiled and the base is not.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/orchestrate.ts, line 664:
<comment>When only the base compile is missing, `columnDelta` now falls through to the raw-SQL path and can emit removed-column warnings. Before this change that case returned no deltas. Confirm this is intended for models marked unverified, or keep returning no deltas when the head is compiled and the base is not.</comment>
<file context>
@@ -648,19 +649,41 @@ async function columnDelta(
+ current,
+ }
+ // Compiled SQL on both sides: column lineage plus the final SELECT's names.
+ if (engineNewSql && (!withOld || engineOldSql)) {
+ const current = await outputColumns(runner, engineNewSql, dialect)
+ const before = withOld ? await outputColumns(runner, engineOldSql!, dialect) : undefined
</file context>
There was a problem hiding this comment.
Fixed in e77d121. A {% if %} or {% for %} tag in the raw select list now makes the list unknown, so conditional columns are no longer reported as added or removed. We kept the raw fallback when only one side is compiled: both sides are then read from raw SQL, so the comparison stays like for like.
| */ | ||
| function dedupWindows(c: Ctx): Array<{ spec: string; text: string }> { | ||
| const sql = stripComments(c.newSql) | ||
| const added = c.added.map((l) => stripComments(l).trim()).filter((l) => l.length >= 3) |
There was a problem hiding this comment.
P2: The length >= 3 filter drops valid added SQL lines such as id or x. A multiline window changed only by adding such a two-character expression is then ignored, so the no-ORDER BY detector misses the change; filter only empty normalized lines.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 235:
<comment>The `length >= 3` filter drops valid added SQL lines such as `id` or `x`. A multiline window changed only by adding such a two-character expression is then ignored, so the no-`ORDER BY` detector misses the change; filter only empty normalized lines.</comment>
<file context>
@@ -225,15 +225,42 @@ const detectNotIn: Detector = (c) => {
+ */
+function dedupWindows(c: Ctx): Array<{ spec: string; text: string }> {
+ const sql = stripComments(c.newSql)
+ const added = c.added.map((l) => stripComments(l).trim()).filter((l) => l.length >= 3)
+ const out: Array<{ spec: string; text: string }> = []
+ for (const m of sql.matchAll(/row_number\s*\(\s*\)\s*over\s*\(/gi)) {
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. When the diff has hunk headers (the PR API always sends them), matching now uses line numbers and no longer depends on the length >= 3 text filter.
| const arg = (t: TestRef, name: string) => t.args?.[name] ?? t.args?.arguments?.[name] | ||
| const shortName = (test: string) => test.split(".").pop()!.toLowerCase() | ||
| const refsEntity = (v: unknown, entity: string) => | ||
| typeof v === "string" && new RegExp(`ref\\(\\s*['"]${entity.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}['"]\\s*\\)`).test(v) |
There was a problem hiding this comment.
P2: refsEntity matches ref( inside names such as my_ref(, creating false self-comparison findings. Require a word boundary before ref and allow normal whitespace before (.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/test-adequacy.ts, line 69:
<comment>`refsEntity` matches `ref(` inside names such as `my_ref(`, creating false self-comparison findings. Require a word boundary before `ref` and allow normal whitespace before `(`.</comment>
<file context>
@@ -0,0 +1,214 @@
+const arg = (t: TestRef, name: string) => t.args?.[name] ?? t.args?.arguments?.[name]
+const shortName = (test: string) => test.split(".").pop()!.toLowerCase()
+const refsEntity = (v: unknown, entity: string) =>
+ typeof v === "string" && new RegExp(`ref\\(\\s*['"]${entity.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}['"]\\s*\\)`).test(v)
+
+function finding(file: ChangedFile, model: string | undefined, rule: string, severity: Finding["severity"], title: string, body: string, extra = ""): Finding {
</file context>
| typeof v === "string" && new RegExp(`ref\\(\\s*['"]${entity.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}['"]\\s*\\)`).test(v) | |
| typeof v === "string" && new RegExp(`\\bref\\s*\\(\\s*['"]${entity.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}['"]\\s*\\)`).test(v) |
There was a problem hiding this comment.
Follow-up: low priority, tracked (\bref).
| const inherited = inheritedThresholds(project, segmentsOf(file.path, input.layout)) | ||
| const own = (t: TestRef) => { | ||
| const c = t.args?.config ?? {} | ||
| return [c.severity, c.error_if, c.warn_if, t.args?.severity, t.args?.error_if].some((v) => v !== undefined) |
There was a problem hiding this comment.
P2: own treats any explicit threshold as overriding every inherited setting, so tests can inherit an ineffective severity or error_if without being reported. Track severity and error thresholds independently.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/test-adequacy.ts, line 164:
<comment>`own` treats any explicit threshold as overriding every inherited setting, so tests can inherit an ineffective `severity` or `error_if` without being reported. Track severity and error thresholds independently.</comment>
<file context>
@@ -0,0 +1,214 @@
+ const inherited = inheritedThresholds(project, segmentsOf(file.path, input.layout))
+ const own = (t: TestRef) => {
+ const c = t.args?.config ?? {}
+ return [c.severity, c.error_if, c.warn_if, t.args?.severity, t.args?.error_if].some((v) => v !== undefined)
+ }
+ const loose = added.filter((t) => !own(t))
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. We'll check each own setting separately.
| data_tests: | ||
| - not_null${own} | ||
| ` | ||
| test("the effective value walks project → folders, deeper wins", () => { |
There was a problem hiding this comment.
P3: The test name claims 'deeper wins', but no case sets the same key at two levels. Add a fixture where a folder overrides the project-level +severity (for example, shop: +severity: error with marts: +severity: warn) and assert the deeper value and its path.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/test/altimate/review-test-adequacy.test.ts, line 56:
<comment>The test name claims 'deeper wins', but no case sets the same key at two levels. Add a fixture where a folder overrides the project-level `+severity` (for example, `shop: +severity: error` with `marts: +severity: warn`) and assert the deeper value and its path.</comment>
<file context>
@@ -0,0 +1,83 @@
+ data_tests:
+ - not_null${own}
+`
+ test("the effective value walks project → folders, deeper wins", () => {
+ const doc = YAML.parse(project)
+ expect(inheritedThresholds(doc, ["marts"])).toEqual({ severity: "warn", path: ["shop", "marts"] })
</file context>
There was a problem hiding this comment.
Follow-up: nit, tracked. We'll add a fixture where the deeper level wins.
| The action can build these for you with `dbt_prepare` (off by default): | ||
|
|
||
| ```yaml | ||
| - uses: AltimateAI/altimate-code/github/review@main |
There was a problem hiding this comment.
P3: This snippet pins the action to @main, but the rest of the page pins @v0.8.5. Pin to the same release tag so users get a reviewed, reproducible action version.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 215:
<comment>This snippet pins the action to `@main`, but the rest of the page pins `@v0.8.5`. Pin to the same release tag so users get a reviewed, reproducible action version.</comment>
<file context>
@@ -209,12 +209,30 @@ jobs:
+The action can build these for you with `dbt_prepare` (off by default):
+
+```yaml
+ - uses: AltimateAI/altimate-code/github/review@main
+ with:
+ dbt_prepare: compile # none | compile | full
</file context>
| - uses: AltimateAI/altimate-code/github/review@main | |
| - uses: AltimateAI/altimate-code/github/review@v0.8.5 |
There was a problem hiding this comment.
Not changing yet: @main is intentional until a release that includes dbt_prepare ships, and we'll re-pin it then.
| const t = thresholds(n) | ||
| if (t.severity) eff.severity = t.severity | ||
| if (t.error_if) eff.error_if = t.error_if | ||
| if (t.severity || t.error_if) path = at.length ? at : ["(all tests)"] |
There was a problem hiding this comment.
P3: path records only the last node containing either setting, so findings misidentify the source when severity and error_if are inherited from different levels. Track a path per threshold.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/test-adequacy.ts, line 117:
<comment>`path` records only the last node containing either setting, so findings misidentify the source when `severity` and `error_if` are inherited from different levels. Track a path per threshold.</comment>
<file context>
@@ -0,0 +1,214 @@
+ const t = thresholds(n)
+ if (t.severity) eff.severity = t.severity
+ if (t.error_if) eff.error_if = t.error_if
+ if (t.severity || t.error_if) path = at.length ? at : ["(all tests)"]
+ }
+ return { ...eff, path }
</file context>
There was a problem hiding this comment.
Follow-up: nit, tracked. Each threshold will track its own path.
| function rawSelectList(sql: string): Set<string> { | ||
| const plain = sql | ||
| .replace(/\{#[\s\S]*?#\}/g, " ") | ||
| .replace(/\{%[\s\S]*?%\}/g, " ") |
There was a problem hiding this comment.
WARNING: rawSelectList blanks Jinja tags but keeps block bodies, so conditional columns become unconditional
{% if %}/{% for %} bodies stay in the text, so select id, amount {% if target.name == 'prod' %}, _loaded_at{% endif %} from t parses as {id, amount, _loaded_at} even when the compiled head renders the false branch. Line 664 now falls back to this raw path whenever either compiled side is missing (e.g. the common "compile the head only, no target-base" setup), so this yields a false columnDelta — a spurious new column that drives on_schema_change, contract, and consumer-list findings, where the old code returned none. Consider treating a model whose select list contains any {% if/for/set %} block as unknown (empty set), and/or using raw only when neither side has compiled SQL.
There was a problem hiding this comment.
Fixed in e77d121. Same fix as r4626962305.
| } | ||
| if (close < 0) continue | ||
| const text = sql.slice(m.index!, close + 1) | ||
| if (!added.some((l) => text.includes(l) || l.includes(text.replace(/\s+/g, " ")))) continue |
There was a problem hiding this comment.
WARNING: "Added line falls inside the window" is a raw substring match, so untouched windows are flagged
text.includes(l) has no line anchoring, so a short or clause-shaped added line elsewhere in the file matches any window that merely contains that text. Example: a pre-existing row_number() over (partition by country, id) (no ORDER BY) is untouched, and the PR adds , id to the select list — text.includes(", id") is true and dedup_no_tiebreaker fires on code the PR never touched, with evidence.line pointing at the pre-existing window. Same for a shared partition by customer_id / order by _loaded_at line. The "a window the PR doesn't touch is not reported" test only covers a line that matches nothing. Match by line range (compute the window's line span and check an added line number falls in it) instead of substring containment.
There was a problem hiding this comment.
Fixed in e77d121. Same fix as r4626962384.
| fi | ||
| if [[ -z "$MB" ]]; then echo "::warning::Could not find the merge-base with $BASE; skipping the base compile."; exit 0; fi | ||
| WT="$RUNNER_TEMP/altimate-review-base" | ||
| git worktree add --detach "$WT" "$MB" >/dev/null |
There was a problem hiding this comment.
WARNING: Unguarded git worktree add aborts the whole review
The base compile at line 227 is best-effort (|| echo "::warning::…"), but git worktree add is a plain command under set -e. If it fails — e.g. a stale $RUNNER_TEMP/altimate-review-base on a reused/self-hosted runner, or a lazy-fetch error for $MB — the step fails and the following "Run dbt PR review" step is skipped, contradicting the documented "a base compile that fails is a warning: the review still runs". Clean $WT first and guard the add (|| { echo "::warning::…"; exit 0; }), and register the cleanup trap before the add.
There was a problem hiding this comment.
Fixed in e77d121. The cleanup trap is set first, and a failed git worktree add now warns and skips the base compile.
| for (const [path, t] of flat(newDoc)) { | ||
| const prev = before.get(path) ?? {} | ||
| const toWarn = t.severity === "warn" && prev.severity !== "warn" | ||
| const newThreshold = t.error_if && t.error_if !== "!= 0" && t.error_if !== prev.error_if |
There was a problem hiding this comment.
WARNING: error_if "loosening" has no direction check and matches the default by exact string
newThreshold = t.error_if && t.error_if !== "!= 0" && t.error_if !== prev.error_if fires for any change, including a tightening: >100 → >10 is reported as "test thresholds loosened … so those tests stop failing the build", the opposite of what happened. It also treats only the exact spelling "!= 0" as the strict default — "!=0", ">0", " > 0 " are all flagged as custom thresholds. Parse the bound and compare direction (only report when the new bound allows more failures), and normalize the strict-default forms.
There was a problem hiding this comment.
Fixed in e77d121. Same fix as r4626962408.
| "equality", | ||
| "equal_rowcount", | ||
| "cardinality_equality", | ||
| "fewer_rows_than", |
There was a problem hiding this comment.
WARNING: Some SELF_COMPARE entries are not "passes by construction"
fewer_rows_than compares row counts with <; against its own ref it always fails (count < count is false), so the emitted body ("passes by construction") is wrong. And expect_table_aggregation_to_equal_other_table is designed to compare two different aggregations: the same model with expression: sum(subtotal) vs compare_expression: sum(net) + sum(tax) is a valid cross-column invariant, yet it is flagged. Only treat aggregation tests as self-comparisons when the expressions match, and drop/adjust fewer_rows_than.
There was a problem hiding this comment.
Fixed in e77d121. Same fix as r4626962291.
| build "$PROJECT" "$PROJECT/target" | ||
| if [[ "$IN_PREPARE" == "full" ]]; then | ||
| echo "Generating catalog.json…" | ||
| ( cd "$PROJECT" && "${DBT[@]}" docs generate --target-path "$PROJECT/target" ${PROFILES[@]+"${PROFILES[@]}"} ) |
There was a problem hiding this comment.
SUGGESTION: dbt docs generate is unguarded in full mode
In full mode this plain command under set -e aborts the step (and skips the review) if the warehouse is unreachable or credentials are missing — exactly the failure the input description warns about. dbt compile at line 207 already produced a usable target/, so degrade instead: … docs generate … || echo "::warning::docs generate failed; continuing without catalog.json.".
There was a problem hiding this comment.
Fixed in e77d121. A failed dbt docs generate now warns and the review continues without catalog.json.
| const rel = root && file.startsWith(root) ? file.slice(root.length) : file | ||
| const parts = rel.split("/").slice(0, -1) | ||
| const modelDirs = layout?.dirs.model ?? ["models"] | ||
| return modelDirs.includes(parts[0]) ? parts.slice(1) : parts |
There was a problem hiding this comment.
SUGGESTION: segmentsOf doesn't normalize multi-segment/relative model-paths
modelDirs.includes(parts[0]) matches only the first segment exactly. A valid model-paths: ["./models"] or ["transform/models"] isn't stripped (parts[0] is models/transform), so inheritedThresholds walks keys that don't exist and misses inherited severity/error_if. roleFromLayout in diff-filter.ts already normalizes these dirs; reuse that (normalize and strip the longest matching configured prefix).
There was a problem hiding this comment.
Follow-up: low priority, tracked. Same as r4626962396.
| [[ -n "${IN_PROFILES:-}" ]] && PROFILES=(--profiles-dir "$(cd "$PROJECT" && cd "$IN_PROFILES" && pwd)") | ||
| build() { # $1 = project dir, $2 = target path | ||
| ( cd "$1" | ||
| [[ -f packages.yml || -f dependencies.yml ]] && "${DBT[@]}" deps ${PROFILES[@]+"${PROFILES[@]}"} |
There was a problem hiding this comment.
SUGGESTION: "${DBT[@]}" expands an array that can be empty
read -ra DBT <<< "$IN_DBT" produces an empty array when dbt_command is empty/whitespace. Under set -u on bash < 4.4 (e.g. macOS 3.2) "${DBT[@]}" is an unbound-variable error; on newer bash the command word vanishes and deps/compile fails obscurely. Validate it ([[ ${#DBT[@]} -gt 0 ]] || { echo "::error::dbt_command is empty"; exit 1; }) and use the file's own ${DBT[@]+"${DBT[@]}"} idiom for consistency.
There was a problem hiding this comment.
Not changing: dbt_command defaults to dbt, and an empty value is a configuration error that fails loudly.
| // A dbt_project.yml change that loosens test thresholds for a whole path. | ||
| for (const file of reviewable.filter((f) => f.kind === "project_config" && /(^|\/)dbt_project\.ya?ml$/.test(f.path))) { | ||
| if (file.status !== "modified") continue | ||
| const [oldContent, newContent] = await Promise.all([getContent?.(file.path, "old"), getContent?.(file.path, "new")]) |
There was a problem hiding this comment.
SUGGESTION: Re-fetch of dbt_project.yml is redundant and unguarded
When the PR modifies dbt_project.yml, this re-reads the head content that projectYml already holds (fetched with .catch at line 1671), and unlike that read it has no .catch. A rejecting getContent (valid custom/unit-test resolver) aborts the entire runReview instead of degrading this lane. Reuse projectYml when file.path === projectYmlPath, and guard the old-side read.
There was a problem hiding this comment.
Follow-up: low priority, tracked. We'll reuse projectYml and add a .catch.
| const out = detectTestAdequacy({ file, newContent: newTests(), projectYml: project }) | ||
| expect(rule(out)).toEqual(["test-inherits-warn"]) | ||
| expect(out[0].body).toContain("`shop` → `marts`") | ||
| const own = newTests(":\n config:\n severity: error").replace("- not_null:", "- not_null:") |
There was a problem hiding this comment.
SUGGESTION: No-op .replace and a silent negative assertion
.replace("- not_null:", "- not_null:") replaces a string with itself (dead code — the interpolated argument already supplies the colon). Also, if a future indentation slip made this YAML invalid, parse() returns undefined, detectTestAdequacy returns [], and line 68 would pass silently. Drop the .replace and assert the parsed shape so the negative assertion is trustworthy.
There was a problem hiding this comment.
Follow-up: nit, tracked. We'll remove the no-op .replace.
Code Review SummaryStatus: 16 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (10 files)
Fix these issues in Kilo Cloud Previous Review Summary (commit 7da6850)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 7da6850)Status: 12 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (10 files)
Reviewed by deepseek-v4.1-flash-1017 · Input: 140.6K · Output: 46.1K · Cached: 2.3M Review guidance: REVIEW.md from base branch |
hkaltai
left a comment
There was a problem hiding this comment.
Verdict: changes requested (one MAJOR). Panel: Claude, Codex, and a lead review. Counts: 0 CRITICAL, 1 MAJOR, 5 MINOR, 4 NIT.
The opt-in dbt_prepare, test-adequacy rules, parse-only column sets and multi-line dedup windows are well structured and tested (targeted suites pass, inputs reach the script only through env, worktree cleanup is trapped). The MAJOR is that the new dedup-window attribution uses a substring match and can flag windows the PR never touched (a regression versus the previous added-line-only behaviour). The MINORs are precision edges in the new threshold and select-list rules and in the action's head handling.
Scope: no linked ticket, so scope vs ticket cannot be verified; the diff matches the PR description. The example workflow rename is a small related cleanup.
Nits
packages/opencode/src/altimate/review/orchestrate.ts(comment beforerawSelectListuse): says "CI that only runsdbt parse" but the fallback also runs when the head is compiled and the base is not. Update the comment.orchestrate.ts:684: the comment "A Jinja expression inside an item ... leaves its alias intact" sits after the guard it explains; move it aboveif (!names || names.includes("__jinja__")).test-adequacy.ts:162const own = .../loose:loosemeans "tests without their own threshold";inheritingis clearer.thresholds()also returnsseverity, sotestConfigLevelswould fit; rule idtest-thresholds-loosenedis also emitted for tightenings.github/review/action.yml:196read -ra DBT <<< "$IN_DBT": ignores quoting and an emptydbt_commandyields an empty array (script would runcompile). Validate non-empty. Alsotest-adequacy.tsown()treats a lonewarn_ifas overriding an inheritedseverity: warn.
| } | ||
| if (close < 0) continue | ||
| const text = sql.slice(m.index!, close + 1) | ||
| if (!added.some((l) => text.includes(l) || l.includes(text.replace(/\s+/g, " ")))) continue |
There was a problem hiding this comment.
[MAJOR] Dedup window is attributed to the PR by substring match, so untouched windows are flagged
if (!added.some((l) => text.includes(l) || l.includes(text.replace(/\s+/g, " ")))) continuetext.includes(l) is a substring test, not a position test. Scenario (reproduced by two reviewers): the model already has an unchanged row_number() over (partition by customer_id) as rn; the PR only adds a customer_id to a group by elsewhere. The added line customer_id is a substring of the window, so dedup_no_tiebreaker fires (high confidence) on code the PR did not touch. Before this PR only added lines that themselves contained row_number were considered. The existing "untouched window" test uses select 1 as unrelated, which can never match.
Fix: map each window's [m.index, close] to new-file line numbers and intersect with the added hunk line numbers (or at minimum require the added line to contain row_number/partition by/order by). Add a test with an unrelated added bare identifier that also appears in the window. Also normalise whitespace on both sides of the second comparison.
| // No compiled SQL (CI that only runs `dbt parse`): the raw model's own SELECT | ||
| // list, when it names its columns. Columns produced inside a macro stay unknown. | ||
| if (!newSql) return none | ||
| const current = rawSelectList(newSql) |
There was a problem hiding this comment.
[MINOR] Parse-only select list treated as complete when items are dropped
const current = rawSelectList(newSql)selectListNames is deliberately conservative (items it cannot name contribute nothing). Previously that only supplemented lineage; here it is the whole answer. select id, amount::numeric(18,2) yields ["id"], so going from amount to a cast is seen as a removed column, and with exposures/contracts this produces a high-confidence "removed columns" warning. The branch also now runs when the head is compiled but the base is not.
Fix: return unknown (empty set) when any non-empty select item produced no name. Related: stripping {% if %} / {% for %} tags keeps their bodies, so {% if var('x', false) %}, region {% endif %} is counted as an unconditional added column and can trigger the on_schema_change='fail' warning; treat projection-affecting control blocks as unknown.
| for (const [path, t] of flat(newDoc)) { | ||
| const prev = before.get(path) ?? {} | ||
| const toWarn = t.severity === "warn" && prev.severity !== "warn" | ||
| const newThreshold = t.error_if && t.error_if !== "!= 0" && t.error_if !== prev.error_if |
There was a problem hiding this comment.
[MINOR] Any changed error_if other than != 0 is reported as "loosened"
const newThreshold = t.error_if && t.error_if !== "!= 0" && t.error_if !== prev.error_ifChanging '>1000' to '>10' (a tightening) or adding the default-equivalent '>0' / '!=0' is reported as "test thresholds loosened". The same literal comparison is used for the inherited-threshold rule. Also, the comparison is per node rather than effective value: with root +severity: warn, deleting marts: +severity: error loosens every marts test but reports nothing, and a redundant +severity: warn under an already-warn parent is reported.
Fix: normalise op N and compare numerically, and compare inheritedThresholds(old, path) with inheritedThresholds(new, path) across the union of paths.
| const project = parse(input.projectYml) | ||
| if (project && added.length) { | ||
| const inherited = inheritedThresholds(project, segmentsOf(file.path, input.layout)) | ||
| const own = (t: TestRef) => { |
There was a problem hiding this comment.
[MINOR] Any local threshold suppresses all inherited checks
const own = (t: TestRef) => {With project-level severity: warn, adding only config: {warn_if: '>10'} or error_if: '!= 0' returns no finding, yet severity is still inherited (dbt resolves these as separate properties).
Fix: merge local and inherited properties individually and inspect the effective config.
| const SELF_COMPARE = new Set([ | ||
| "equality", | ||
| "equal_rowcount", | ||
| "cardinality_equality", |
There was a problem hiding this comment.
[MINOR] cardinality_equality / aggregation tests against the same model are not always self-comparisons
"cardinality_equality",cardinality_equality with to: ref('m'), field: b on column a compares two different columns' distinct counts, a real check, but is reported as comparing the model with itself. Same for expect_table_aggregation_to_equal_other_table with different expressions. (fewer_rows_than against itself fails rather than passes, so "passes by construction" is also inaccurate there.)
Fix: for these tests require the field/expression to match as the relationships branch does, and drop fewer_rows_than. Separately, segmentsOf strips only model dirs, so seeds/_s.yml walks data_tests: shop: seeds: as if seeds were a folder; strip seed/snapshot dirs too.
| "${DBT[@]}" compile --target-path "$2" ${PROFILES[@]+"${PROFILES[@]}"} ) | ||
| } | ||
| echo "Compiling the head…" | ||
| build "$PROJECT" "$PROJECT/target" |
There was a problem hiding this comment.
[MINOR] dbt_prepare compiles the checkout, ignoring the head input; head-side failures abort the job
build "$PROJECT" "$PROJECT/target"- The merge-base is computed against literal
HEADand the head compile uses the checkout, but the review step passes--head "$IN_HEAD"when set. Withhead: Awhile B is checked out, A's diff is reviewed using B's compiled SQL and equivalence can be wrongly reported. Prepare against the selected head or reject a mismatch. - Under
set -e, a failed headdeps/compile/docs generatefails the step so the review never runs (even inmode: comment), while the input description and docs (docs/docs/usage/dbt-pr-review.md:224) say compile "usually works offline". Many adapters need a connection to compile. Either degrade to a warning like the base side, or document that it fails the job and drop the offline claim. - Minor:
manifest_pathis not honoured (head always goes to$PROJECT/target), and a runtime-generatedprofiles.ymlin the project dir is absent from the base worktree, so the base compile quietly fails.
…erge-base and catalog artifacts - new action inputs: `dbt_prepare` (`none` default | `compile` | `full`), `dbt_command`, `dbt_profiles_dir` - `compile` runs `dbt compile` on the head and compiles the PR's merge-base into `target-base/` in a temporary worktree (fetching and deepening a shallow checkout when needed); `full` adds `dbt docs generate` - a base compile that fails is a warning: the review still runs and marks those models unverified - inputs reach the script only through env, never interpolated - docs: the manual base-compile snippet now compiles the merge-base, not the base branch's tip - the example workflow is renamed `examples/dbt-pr-review.yml` (was named after one internal repo) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ows split across lines are read whole Found by replaying the PRs behind known escaped defects: - without compiled SQL, output columns come from the raw model's final SELECT (Jinja blanked; a Jinja expression in the list makes it unknown). `on_schema_change='fail'` with new columns now fires on the two real escapes it targets, in a parse-only CI setup - `row_number() over (…)` windows are read from the whole file, so an `ORDER BY` on its own line is seen: the "dedup has no ORDER BY" rule no longer fires falsely on multi-line windows, and the load-timestamp rule sees them. A window counts only when an added line falls inside it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nherited warn/thresholds, loosened project thresholds - a test added or changed in the PR that compares a model with itself (`dbt_utils.equality`/`equal_rowcount`/… against its own `ref`, or a `relationships` test to its own column) passes by construction (warning) - new tests that inherit `severity: warn` or an `error_if` threshold from `dbt_project.yml` (walked from the top through the project name and folders, deeper wins) are reported once per yml (suggestion) - a `dbt_project.yml` change that turns a folder's tests to `warn` or sets an `error_if` threshold is reported (warning) - reads dbt 1.8+ `arguments:` too; all findings are catalog origin (never block) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…window touch, self-compare tests, `error_if` loosening, prepare-step failures
- `orchestrate.ts`: a `{% if %}` / `{% for %}` tag inside a raw model's select
list makes the list unknown, so conditional columns no longer read as added
or removed in the parse-only column delta
- `dbt-patterns.ts`: a dedup window counts as changed only when an added line
falls inside its line span (head line numbers from the hunk headers), not
when an added line's text happens to occur in it
- `test-adequacy.ts`: drop `fewer_rows_than` (it fails against itself, it
doesn't pass by construction); an aggregation comparison on the same model
is flagged only when both expressions match
- `test-adequacy.ts`: `error_if` is reported as loosened only when the new bound
tolerates more failing rows; `> 0` / `!=0` / `>= 1` count as the strict default
- `action.yml`: a failed `dbt docs generate` or base checkout warns instead of
failing the step; a stale `target-base/` is removed before the base compile
- tests for each
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fe31dc6 to
9625cfd
Compare
7da6850 to
e77d121
Compare
| trap 'git worktree remove --force "$WT" >/dev/null 2>&1 || true' EXIT | ||
| git worktree add --detach "$WT" "$MB" >/dev/null || { echo "::warning::Could not check out the base; skipping the base compile."; exit 0; } | ||
| # A stale target-base/ from an earlier run must not stand in for a failed compile. | ||
| rm -rf "$PROJECT/target-base" |
There was a problem hiding this comment.
WARNING: The stale target-base/ cleanup runs only after the guards that can skip the base compile
When BASE is empty, the merge-base can't be found, or git worktree add fails, the script exits 0 before this line, so a target-base/ left by an earlier run survives. makeCompiledResolver reads target-base/compiled whenever it exists (compiled.ts:97), so on a cached or self-hosted workspace the head is compared against the previous run's base, and because the old side still resolves, those models are not even listed as unverified. Move the cleanup above the BASE/MB/worktree guards, and make it non-fatal (rm -rf ... || { echo "::warning::..."; exit 0; }) so an rm failure cannot abort the step and skip the review.
| } | ||
|
|
||
| /** Head-side line numbers of the added lines; undefined when the diff has no hunk headers. */ | ||
| function addedLineNumbers(diff: string | undefined): Set<number> | undefined { |
There was a problem hiding this comment.
SUGGESTION: This hunk walker duplicates the exported commentableLines
commentableLines (orchestrate.ts:268) already walks @@ hunks with the same regex and the same right counter to compute head-side line numbers; this copy already differs (it does not drop the diff's trailing-newline row) and the two will drift. dbt-patterns cannot import orchestrate without a cycle (the file already keeps a local modelNameFromPath for exactly that reason), so extract the walker into a shared module (e.g. diff-filter.ts) and take the added-line set from it, or pass the numbers in through Ctx.
| */ | ||
| function dedupWindows(c: Ctx): Array<{ spec: string; text: string }> { | ||
| // Comments blanked with newlines kept, so offsets map to head line numbers. | ||
| const sql = c.newSql.replace(/\/\*[\s\S]*?\*\/|--[^\n]*|\{#[\s\S]*?#\}/g, (m) => m.replace(/[^\n]/g, " ")) |
There was a problem hiding this comment.
SUGGESTION: This comment blanking is not quote-aware; blankComments() already does this
A -- inside a string literal blanks the rest of the line: select '--' as sep, row_number() over (partition by id) as rn loses the window entirely (a missed dedup finding), and a /* in a literal can blank a multi-line span. blankComments() in select-list.ts is the existing length-preserving, quote-aware helper (it skips '...', "..." and backticked identifiers), so reuse it here and keep only the {# ... #} alternative.
| const run = prep.run! | ||
| expect(run).toMatch(/docs generate[^\n]*\) \|\|\n\s*echo "::warning::/) | ||
| expect(run).toMatch(/git worktree add[^\n]*\|\| \{ echo "::warning::/) | ||
| expect(run.indexOf('rm -rf "$PROJECT/target-base"')).toBeLessThan(run.indexOf('build "$WT/$REL"')) |
There was a problem hiding this comment.
SUGGESTION: This ordering assertion passes when the cleanup line is absent
run.indexOf('rm -rf ...') is -1 when the line is deleted, and -1 < indexOf('build ...') is true, so the test still passes without the rm -rf "$PROJECT/target-base" line it is meant to guard. Assert the cleanup line is present (e.g. expect(run).toContain('rm -rf "$PROJECT/target-base"')) before comparing indexes.
Issue for this PR
Part of #1792
Stacked on #1725 (← #1724 ← #1723 ← … ← #1719); merge in that order. Draft.
Type of change
What does this PR do?
This closes the remaining acceptance-criteria gaps in the review stack. Two of the fixes came from replaying the PRs behind known escaped defects.
dbt_prepare). Defaultnone, so nothing changes for existing users.compilerunsdbt compileon the head, then compiles the PR's merge-base intotarget-base/in a temporary worktree. A shallow checkout is fetched and deepened when needed.fullalso runsdbt docs generate.equality/equal_rowcountagainst its ownref, or arelationshipstest to its own column) passes by construction;severity: warnor anerror_ifthreshold fromdbt_project.ymlare reported;dbt_project.ymlchange that loosens those thresholds for a folder is reported.dbt parse-only CI. Without compiled SQL, output columns come from the raw model's final SELECT; a Jinja expression in the list makes it unknown. In a parse-only replay,on_schema_change='fail'with new columns now fires on the two real escapes it targets.order byis on its own line, and the load-timestamp rule sees such windows.How did you verify your code works?
review-action.test.ts:dbt_prepareis opt-in, reads inputs only from env, and builds intotarget-base/.review-test-adequacy.test.ts: self-comparison, inheritedwarn/error_if, loosened thresholds, dbt 1.8+arguments:.review-incremental-semantics.test.ts: multi-line windows, raw-SQL column delta.bun typecheckis clean, and the real-world corpus is unchanged (15/15 caught, 0/5 false positives).dbt_preparerun locally against a real dbt 1.10 DuckDB project. The head compiled intotarget/, the merge-base compiled intotarget-base/(the base still has the removed column), and the temporary worktree was cleaned up.on_schema_changenow catches two of them (two escapes each).Cost side effects
dbt_prepare: compileadds onedbt compileper side of the PR in CI, andfulladdsdbt docs generate(warehouse queries for the catalog). Both are opt-in; the defaultnoneadds nothing. No model calls.Screenshots / recordings
N/A (CLI / action).
Checklist
🤖 Generated with Claude Code
Summary by cubic
Closes the remaining acceptance-criteria gaps in the dbt review stack: adds an opt-in
dbt_prepareaction input for building the artifacts the full review needs, and fixes two classes of missed findings — data tests that can't do their job, and column-set rules that skipped work in parse-only CI.New
dbt_prepareaction inputnone;compilerunsdbt compileon the head and the PR's merge-base intotarget-base/, andfullalso runsdbt docs generate.target-base/is removed before the base compile, docs now compile the merge-base, and the example workflow gets a generic name.Review gap fixes
equality/equal_rowcount/relationshipsorexpect_table_aggregation_to_equal_other_tablewith matching expressions, plus tests that inheritseverity: warnor anerror_ifthreshold fromdbt_project.yml.dbt_project.ymlchanges that loosen those thresholds, counting only bounds that tolerate more failing rows.on_schema_change='fail'fires in parse-only CI; a Jinja expression or{% if %}tag in the list leaves it unknown.ORDER BYno longer triggers a false "no ORDER BY".Written for commit e77d121. Summary will update on new commits.